feat(lance): add Lance REST server packaging by beinan · Pull Request #8825 · apache/gravitino

beinan · 2025-10-17T02:24:09Z

What changes were proposed in this pull request?

add the new lance-common and lance-rest-server modules that bootstrap a minimal Lance catalog REST facade backed by in-memory state
introduce REST resources (LanceCatalogOperations, LanceMetadataOperations) plus a GravitinoLanceRESTServer entry point and auxiliary service wiring
package assets for the Lance REST server (config + startup script) and hook the module into Gradle distribution/standalone tarball tasks with checksums
extend build dependencies (SLF4J, Dropwizard metrics, Prometheus bridge) so the service runs using existing Gravitino telemetry components
surface scaffolding build.gradle.kts under lance/ to disable blanket task execution and ensure module resolution

Why are the changes needed?

the Lance integration needs a thin REST façade to iterate on metadata APIs before the backend is available; this scaffolding mirrors the Iceberg REST
flow
distribution packaging must include the new service so it can be assembled, tested, and deployed alongside other Gravitino auxiliaries

Fix: #N/A

Does this PR introduce any user-facing change?

new Lance REST server tarball and config/script templates shipped with the distribution
new REST endpoints under /lance/v1/... exposing catalog metadata (namespaces, tables)
no breaking changes to existing services

How was this patch tested?

./gradlew :lance:lance-rest-server:build
started the service locally with java -cp lance/lance-rest-server/build/libs/* ...GravitinoLanceRESTServer /tmp/gravitino-lance-rest-server.conf
curl http://127.0.0.1:9101/lance/v1/catalog
curl http://127.0.0.1:9101/lance/v1/catalog/namespaces
curl "http://127.0.0.1:9101/lance/v1/metadata/table?namespace=default&name=sample_table"
curl -X POST http://127.0.0.1:9101/lance/v1/catalog/namespaces -H 'Content-Type: application/json' -d '{"namespace":"demo"}'
curl -X DELETE http://127.0.0.1:9101/lance/v1/catalog/namespaces/demo

jerryshao · 2025-10-17T04:19:03Z

Hi @beinan I created a new branch for Lance development. Can you please create a PR against that branch (https://github.com/apache/gravitino/tree/dev-lance-namespace)?

beinan · 2025-10-17T06:05:56Z

Hi @beinan I created a new branch for Lance development. Can you please create a PR against that branch (https://github.com/apache/gravitino/tree/dev-lance-namespace)?

Yes, I made the PR against the new branch now

jerryshao · 2025-10-17T06:19:14Z

Hi @beinan , The branch setting has some limitations; only the branch name started with "branch-" will run the CI, so I created a new branch (https://github.com/apache/gravitino/tree/branch-lance-namepspace-dev). Can you refresh you PR against new branch?

beinan · 2025-10-18T07:20:07Z

@jerryshao could you help approve the CI? thanks

beinan · 2025-10-19T01:55:39Z

Hey @jerryshao, could you help enable me to start the CI?

yuqi1129 · 2025-10-20T01:57:38Z

We need to cherry-pick @diqiu50fix about CI to branch-lance-namepspace.

jerryshao · 2025-10-20T02:09:24Z

We need to cherry-pick @diqiu50fix about CI to branch-lance-namepspace.

Let me rebase that branch.

### What changes were proposed in this pull request? can specify a principal and authorizer to do authz ### Why are the changes needed? polish authz codes for easier reuse ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? tests updated

mchades · 2025-10-20T07:08:13Z

+  @Path("/{id}/list")
+  public Response listNamespaces(
+      @Encoded @PathParam("id") String namespaceId,
+      @DefaultValue(".") @QueryParam("delimiter") String delimiter,


The Lance namespace spec uses $ as the default delimiter.

Ref: https://lancedb.github.io/lance/format/namespace/concepts/#object-name

good call, changed to $

…ity cache is full. (apache#8816) ### What changes were proposed in this pull request? Adjust the caffeine weight for tags and policies to allow them to evict first compared to other entities. ### Why are the changes needed? They are less important than entities like tables, filesets. Fix: apache#8815 ### Does this PR introduce _any_ user-facing change? N/A. ### How was this patch tested? UTs --------- Co-authored-by: Jerry Shao <jerryshao@datastrato.com>

yuqi1129 · 2025-10-20T12:44:08Z

+  public static final ConfigEntry<String> CATALOG_NAME =
+      new ConfigBuilder(LANCE_CONFIG_PREFIX + "catalog-name")
+          .doc("Logical Lance catalog served by the REST endpoint")
+          .version(ConfigConstants.VERSION_0_1_0)


Will Lance rest plan to use a separate version management system, and the version starts from 0.1.0?

I don't have strong preference, what do you think?

iceberg rest service is using 0.2.0

public static final ConfigEntry<String> CATALOG_BACKEND = new ConfigBuilder(IcebergConstants.CATALOG_BACKEND) .doc("Catalog backend of Gravitino Iceberg catalog") .version(ConfigConstants.VERSION_0_2_0) .stringConf() .createWithDefault("memory");

mchades

lgtm. @jerryshao Would you like to take another review?

jerryshao · 2025-10-21T06:36:21Z

Let's merge it first for the quick iteration.

### What changes were proposed in this pull request? - add the new lance-common and lance-rest-server modules that bootstrap a minimal Lance catalog REST facade backed by in-memory state - introduce REST resources (LanceCatalogOperations, LanceMetadataOperations) plus a GravitinoLanceRESTServer entry point and auxiliary service wiring - package assets for the Lance REST server (config + startup script) and hook the module into Gradle distribution/standalone tarball tasks with checksums - extend build dependencies (SLF4J, Dropwizard metrics, Prometheus bridge) so the service runs using existing Gravitino telemetry components - surface scaffolding build.gradle.kts under lance/ to disable blanket task execution and ensure module resolution ### Why are the changes needed? - the Lance integration needs a thin REST façade to iterate on metadata APIs before the backend is available; this scaffolding mirrors the Iceberg REST flow - distribution packaging must include the new service so it can be assembled, tested, and deployed alongside other Gravitino auxiliaries Fix: #N/A ### Does this PR introduce any user-facing change? - new Lance REST server tarball and config/script templates shipped with the distribution - new REST endpoints under /lance/v1/... exposing catalog metadata (namespaces, tables) - no breaking changes to existing services ### How was this patch tested? - ./gradlew :lance:lance-rest-server:build - started the service locally with java -cp lance/lance-rest-server/build/libs/* ...GravitinoLanceRESTServer /tmp/gravitino-lance-rest-server.conf - curl http://127.0.0.1:9101/lance/v1/catalog - curl http://127.0.0.1:9101/lance/v1/catalog/namespaces - curl "http://127.0.0.1:9101/lance/v1/metadata/table?namespace=default&name=sample_table" - curl -X POST http://127.0.0.1:9101/lance/v1/catalog/namespaces -H 'Content-Type: application/json' -d '{"namespace":"demo"}' - curl -X DELETE http://127.0.0.1:9101/lance/v1/catalog/namespaces/demo --------- Co-authored-by: mchades <liminghuang@datastrato.com> Co-authored-by: Mini Yu <yuqi@datastrato.com> Co-authored-by: Jerry Shao <jerryshao@datastrato.com>

beinan changed the base branch from main to dev-lance-namespace October 17, 2025 05:56

beinan force-pushed the beinan/lance_catalog branch from c753cb1 to ea7a30d Compare October 17, 2025 06:10

beinan changed the base branch from dev-lance-namespace to branch-lance-namepspace-dev October 17, 2025 06:21

jerryshao closed this Oct 17, 2025

jerryshao reopened this Oct 17, 2025

beinan changed the title ~~feat(lance): add Lance REST server packaging (#0)~~ feat(lance): add Lance REST server packaging Oct 17, 2025

beinan force-pushed the beinan/lance_catalog branch from 43b13eb to e91ccb9 Compare October 18, 2025 07:18

mchades reviewed Oct 20, 2025

View reviewed changes

yuqi1129 reviewed Oct 20, 2025

View reviewed changes

beinan added 5 commits October 21, 2025 03:08

feat(lance): add Lance REST server packaging (#0)

1978fd0

[no-issue] feat(lance): expose namespace table listing (#0)

2362341

[no-issue] fix(lance): register namespace REST handlers (#0)

8c3cfd7

[no-issue] feat(lance): add namespace listing DTOs

44f5c64

Change default delimiter

763281b

beinan force-pushed the beinan/lance_catalog branch from 098dfaa to 763281b Compare October 21, 2025 03:08

mchades approved these changes Oct 21, 2025

View reviewed changes

mchades assigned beinan Oct 21, 2025

mchades requested a review from jerryshao October 21, 2025 03:24

jerryshao merged commit eb8ace8 into apache:branch-lance-namepspace-dev Oct 21, 2025
28 checks passed

Conversation

beinan commented Oct 17, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

jerryshao commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beinan commented Oct 17, 2025

Uh oh!

jerryshao commented Oct 17, 2025

Uh oh!

beinan commented Oct 18, 2025

Uh oh!

beinan commented Oct 19, 2025

Uh oh!

yuqi1129 commented Oct 20, 2025

Uh oh!

jerryshao commented Oct 20, 2025

Uh oh!

mchades Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

beinan Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

yuqi1129 Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

beinan Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

beinan Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

mchades left a comment

Choose a reason for hiding this comment

Uh oh!

jerryshao commented Oct 21, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jerryshao commented Oct 17, 2025 •

edited

Loading

yuqi1129 Oct 20, 2025 •

edited

Loading